Dynamic facial expression recognition using a be- havioural model

A recent interest appears in transportation for users emotion recognition. This permits to adapt car behaviors to drivers mood for safety reasons, or improve public transportation offers. Human emotions are complex and defined by several elements, such as voices intonations or facial expressions. We propose a new dynamic facial expression recognition framework based on Discrete Choice Models (DCM). The aim of the work is to model the choice of a person who is exposed to a video sequence representing a facial expression, and has to label it. The approach originality lies on the absence of ground truth and the explicit modelling of causal effects between facial features and face expression. The model is composed of two parts: the first one captures the dynamic facial expression evaluation across the frames in the sequence, and the second one concerns the frames weighting in order to determine at which moment the person decides the facial expression when looking at the video sequence. A computer vision tool, called Active Appearance Model (AAM) is used to extract facial information in videos. Concerning the dynamic expression evaluation, we assume that the person’s perception evolves at regular time intervals (1 second is chosen). For each time interval a utility function is associated with each possible label (happiness, surprise, neutral, fear, anger, disgust, sadness, other, not known) in order to capture the decision maker’s instantaneous perception. It contains some measures about the face in the associated frames according to the Facial Action Coding System (FACS), as well as facial texture attributes (different levels of grey on the face). For the frames weighting, a utility function is associated to each frame and contains information about the frame dynamic, such as derivatives of feature characterising the face. Finally both parts are linked with the observed choice in the construction of the likelihood function. The model is then estimated using videos from the Facial Expressions and Emotions Database (FEED). Expressions labels on the videos have been obtained using an internet survey available at http://transp-or2.epfl.ch/videosurvey/.